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Author's Abstract 



Several studies have been concerned with the definition of structural 
variabies in word problem (problem solving) exercises^in arithmetic with 
the view of identifying those variables which account for the most variance 
in P(correct) in a linear regression analysis. This study attempted to 
determine whether it was practical and possible to prepare a set of word 
problems of a predicted level of difficulty based on the variables and 
regression equations developed in previous work. Four sets of problems 
were given to students in each of Grades and 7~9> eight sets in all. 

The general equation from the previous did not yield accurate pre- 
dictions for the problems in the present study using a chi-square test. 
However » when new equations, based on the old data, were computed for 
each grade level, the predictions were more accurate, though still not 
significant. Although the predictions were not as close as hoped, the 
means of the residuals was only 11 per cent, range 'f-lS per cent. This 
is quite close for a first attempt. The data indicated that the relative 
difficulty of the exercises was nearly the same over grade levels indicating 
that the model should be capable of predicting more accurately, within 
5 per cent, with further refinement. 
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Introduction 



Problem and Objectives 

There is, as yet, no adequate theory for learning mathematics 
that can be used as a basis to predict the rate of learning or the 
relative difficulty of a given set of verbal problem-solving exercises. 
However, the results of a series of studies, using linear regression 
models, seem to indicate that a nmall but manageable set of variables, 
which strongly influence the relative difficulty of verbal problems, have 
been identified and defined in such a way as to pursue a test of their 
adequacy by using them in the preparation of exercises of a specified 
level of difficulty in advance of their being solved by students. 

The objective of this study was to test the set of above-mentioned 
variables by preparing sets of verbal problem-solving exercises having a 
predicted level of difficulty, in terms of a predicted probability correct, 
and to compare the predicted level of difficulty for each set of items and 
each item individually with the actual performance of students in public 
schools classrooms who attempt to solve the exercises. 

The use of linear regression models to predict the relative difficulty 
of a variety of types of exercises including verbal problem solving 
exercises in well documented (5uppes, Jerman, and Brian I968; Suppes, Loftus 
and Jerman, 1969; Jerman, 1971; Suppes and Morni ngstar, 1972) . One of the 
purposes of these regression studies has been to identify anJ quantify in 
a clear and explicit way a set of structural variables that account for a 
significant amount of the variance in the observed error rate. 

A basic assumption of this approach is that the structure of the 
arithmetic problem itself, to a large measure, determines its difficulty 
level. This is not to say that student aptitude-interaction factors 
do not come into play, but until clear evidence is available concerning 
the existence and nature of any such factors, the structural type of 
analysis may prove to be a nwre fruitful avenue for research and curriculum 
development. What is hoped for eventually is to be able to formulate a clear 
set of rules or a formula for generating sets of arithmetic problems of a 
specified difficulty leveK Curricul um developers would then be in a better 
position to control difficul ty level when preparing instructional materials. 

Review of the literature 

The purpose of this section Is to review some of the attempts made to 
identify and define a meaningful set of variables that can account for 
a significant amount of the variance in the difficulty level of problems 
solved correctly. 

The regression model : The notation used here for the regression 
model itself, follows Suppes, Hyman and Jerman (1966) and denote the j 
variable of problem i by v... The weight assigned to the jth variable is 
denoted byoCj. Let p. be trte observed proportion of correct responses on 

1 



problem i for a given group of students. The purpose of the model is to 
predict p. for each problem. The linear regression model in terms of the 
variables v.. and the weights^ • is then 



P: 



J 



ij 



(1) 



This model, as stated, may not preserve probability since the 
estimated weighting and values for the variables are combined to predict p.. 
Therefore, it has been the practice, in order to insure that the predicted ' p. ' s 
will always lie between 0 and 1, to make the following transformation and ' 
define a new variable z.. 



Zj = log 



1 



(2) 



Then the regression model becomes 



J 'J 



(3) 



To take care of the case when the observed p. is either 1 or 
0, the following transformation was used: ' 



iog (2n.-l) for p. = 0 



log 



2n.-l 

I 



for p. = 1, 



where n. is the total number 
reason for putting 1 - pj in 



of students resppndi ng to problem i. The 
I ... the numerator of equation (2) is to make the 
variables z, increase monotonical ly in difficulty. It is desirable that the 
model reflect an increase in difficulty directly rather than inversely as 
the magnitude of the variables v.j increases. 

Development of Variables in a CAI Context . The variables considered 
were many; those tested on a "set of 68 word problems and reported in the 
Suppes, Loftus and Jerman (I969) paper were subsequently modified and 
retested on the same data base. 

The modified set of variables were defined as follows: 



Operations: the minimum number of operations required to reach a 

correct solution (values range 1-A). 
Steps: the minimuin number of binary operations, steps, needed 

to reach a solution (value range 1-7). 
Length: the number of words in the problem 

(value range 7"5l). 
Conversion: this factor is present if a conversion is required and the 

equivalent units are not given in the problem (a 0, 1 variable)^ 
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Verbal cue: the cue for each operation is as folloK's* 

Cue Words 



Operat ion 

Addition: 
Subtract ion : 
Multipl icat ion: 
Di vi s ion: 



added, altogether, gained 
how much less, lost, left 
each 
average 



If a cue word was present, the value was 1, otherwise 0. 
Order: If the steps to solution were in order as given in the 

Problem statement, the value was 1, otherwise 0, 
Formula: If knowledge of a formula was required, the value was 1, 

otherwise 0, 

Average: If the problem statement contained the word "average'* 

the value was 1, otherwise 0, 
Addition: If the problem required addition, the value was 1, 

otherwise 0. 

Subtraction: If the problem required subtraction, the value was 1, 
otherwise 0. 

Multipl icat ion: . If the problem required multiplication, the value 
was 1 , otherwi se 0, 
^ Division: If the problem required division, the value was 1, 
otherwise 0. 

Sequence: If the problem was in unusual order, the value was 1, 
otherwi se 0, 



Three of the problems in the original set of 68 were deleted due to 
their highX values. The above variables were tested on the data from 
the remaining 65 problems. Of the I6 variables in the expanded set, 
12 were entered by the step-wise regression program, BMD02R. • The value 
of the multiple R was .820. 



After studying the weights of the variables, their contribution to 
the total and their definitions, three additional variables were for 
mulated. Two of these, and S^, were sequential variables; the third 
was a memory variable. T 
are as f ol lows. 



2, were sequential variables; the 
def i ni tions "for the three additional variables 



Memory (M) is defined as the sum of: 
C the number of conversions + knowledge of formulas, 
D the number of numerals in the problem statement, 
0 the number of different operations. 

is defined as the number of displacements of order of operations in 
successive problems. 



Exampl es : 

First Problem 3 + 
Second Problems " 

First' Problem (3 
Second Problem (3 



S, = 1 



A) 
A) 



X 

•4- 



} 



First Problem 3 + ; 
Second Problem A + 1 

First Problem (3 + 
Second Probleifi (3 ^« 



A) 



s, = 0 



} 



S, = 2 
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$2 is defined as the number of displacements between order of operations 
required to solve the problem and the order of operations given in the 
problem statement. 

The total R for the last three variables, Memory, S, and S,, when used 
alone, for the set of 65 original problems was .51. Using all 19 variables 
in the stepwise regression produced a multiple-R of .8A2 for the 12 
variables that contributed at least .001 to an increase in r!" Using the 
last three variables did not increase the number of variables entered. 
Rather, the three new variables entered ahead of the others with an Rof 

In addition to the 19 variables described above, the tollowinq were 
formulated and tested. ' ^ 

Operations 2: The sum of the following, 

1. The number of different operations. 

2. Add ^ if one of .the operations is division. 

Add 2 if one of the operations is multiplication. 
Add 1 if one of the operations is addition. 

Order 2: The sum of the following. 

1. S| 

2. Verbal cue necessary to establish a new order. 

One point for each direct cue missing for each step. 

Recall: The sum of the following. 

1. One count for a formula to be recalled and a count 

for each step in the formula, e.g., A = 21 + 2w (count = 3) 

2. One count for each conversion to be recalled and used. 

3. One count for each fact from a previous problem to be 
recal led and used. 

Verbal Cue 2: The set of cues was expanded. In addition, one count 
vvast,given for each cue present in the problem. 

Addition: added, altogether, gained, total 

Subtraction: how much less, lost, left 
how much larger . . . than 
bow much smaller . . . than 
how much greater . . . than 
how much further . . . than 

Multiplication: each, times 

Division: average 

Distractors: This variable was defined as 1 count for each verbal cue 
which was not a cue for an operation, but a distractor; 
for example. If the word "average" was used but multiplication 
rather than division was the required operation. 
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A complete of variables, by number, follows. 



Variable Name Variable Name 



1 


P (correct-observed) 


12 


Mul t i pi i cat ion 


2 


Operations 


13 


Divi sion 


3 


Steps 


\k 


Sequence 


4 


Length 


15 


s' 

^2 


5 


Convers ions 


16 


6 


Verbal Cue 


17 


Memory 


7 


Order 


18 


Operations-2 


8 


Formula 


19 


Order-2 


9 


Average 


20 


Recal 1 


10 


Addition 


21 


Verbal Cue-2 


]'l 


Subtraction 


22 


Distractor Cue 



When the stepwise regression was applied using these 22 variables 
and the previous 19 variables, there was relatively little gain in 
R and R after the tenth step. In fact, had one considered only those 
variables whose cpntribution to the increase in R^ was .01 or greater, 
the first 8 variables from the set of 19 and the first nine variables from 
the set of 22 would comprise the set of variables of interest. It is 
most interesting to compare the order of entry of the variables in each 
case. The following list may be useful for this purpose. 

22 Variables 

Step R 

1 Operations ^657 

2 Convefi-iGiiv . . /02 

3 Lengtli .7'40 
k Order 2 .7&0 

5 Division .805 

6 $2 .821 

7 Order .829 

8 Memory .835 

9 Distractor Cues.8Al 

Perhaps the most that can be done' at this point, on the basis of 
the analysis thus far, is to note which of the variables (operations, 
length, division, $2 the internal sequence variable, and conversions) 
appears to be the most robust. Memory and distractor cues may or may 
not play important roles in subsequent analyses. 

Analyses of Problem Variables on Paper and Pencil Tests . All previous 
analyses v;ere performed on CAI curriculum where the students indicatjed the 
operations to be performed, but did not actually perform the computations. 
It was of interest to determine if these same variables were applicable to 
problems solved with pencil and paper. Eleven of the 22 variables described 
above were tested on word problems solved off-line to see if their order of 
entry >n the stepwise regression was at all similar to that found in on-line 
CAI context. 



19 Variables 
Step R 

1 Operat ions .657 

2 Verbal Cue .697 

3 Division .729 
k Length .761 

5 Formula .785 

6 .805 

7 Conversions .825 

8 $2 .835 
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The variables selected for testing were the following. 

Variable ^ Name Variable Name 



2** Operations 2 8 Length 

3 Order 2 9 Verbal Cue 

4 Recall 

5 10 Conversion 

6 Memory 11 Formula 

7 S2 12 Division 



These variables were first tested on a collection of problems selected 
for analysis from a set used by average f ifth-grade students in a typical * 
paper-and-penci 1 classroom setting. 

When the stepwise regression, using the set of 11 variables, was 
applied, length, the count of the number of words in the statement of 
the problem, entered first followed by Memory, S«, S^, and Verbal Cue. 
The total R after 9 steps was .77 with r2 = .592,. The variables that 
accounted for most of the variance on-line were also effective, though at 
a lower level, in accounting for much of the variance off-line where 
students were doing the required computation by hand. 

In an attempt to improve the fit, two new variables were defined. 
The first. Verbal Cue 1 (No. 13) was redefined.. It was essentially the 
definition used in Verbal Cue 2, except that it was a 0-1 variable rather 
than a frequency variable as is Verbal Cue 2. The second new variable 
(No. U) was a combination of Verbal Cue 1 and indirect cues, such as "in 
all,*' for addition, "short of . . . ," for subtraction and "per . . . ," 
for mul t ipl ication. The increase in the value of R, upon applying the 
stepwise regression, due to the addition of the two additional variables 
was small, almost .03 from .771 to .799. 

Two additional new variables were added for testing. Verbal Cue 2 
(No. 15) and a distractor variable (No. 16). These two variables are 
the ones described earlier. The value of R obtained as the result of using 
15 variables was .83^ as compared to an R of .799 when 13 variables were 
used. 

Clearly, the fit of the variables selected and tried thus far was less 
than satisfactory. The fact that the off-line students did all their com- 
putation by hand led to the definition of four new computational variables. 
These definitions followed the work reported in Suppes and Morningstar (1972). 
The variables were: 

17. EXMC. A count of 1 was assigned for each multiplication exercise, 
required in the solution of the problem. If multiplication 
was not required, 0 was assigned. 

18. N0MC2. A count of 1 was assigned each time a regouping occurred in 
each multiplication exercise in the problem. For example: 

*Variable 1 was the observed p(correct). 
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/* 

38 
X 5 

190 NOMC = 2 
/ 

38 
X 25 

190 NOMC = 3 
950 

19. C0LC2. For this variable a count of 1 was given for each column and 

a count of 1 was given for each regrouping in addition and 
subtraction exercises. This count applied to only the largest 
exercise in the problem. If no addition or subtraction was 
required, 0 was given. 

20. (JUOT. A count of 1 was given for each digit in the quotient if 

division was required and 0 otherwise. 

Before additional analyses were performed, another problem (p(correct) ^ 
.50) was added to bring the total number of problems up to 30. This set 
of 30 problems were coded on the 19 variables indicated above. 

A regression was run on the 30 problems using all 19 variables. The 
value of R after the first five steps was 0.93, = 0.87. This is a 
surprisingly good fit for just five variables. Perhaps even more surprising 
is the strength of the computational variables as indicated by their point 
of entry into the regression program. Of the first five variables which 
entered the regression, three were computational variables: NOMC, a 
multiplication variable; QUOT, a division variable; and COLC, an addition 
and subtraction variable. The variable LENTH, which accounted for the number 
of words in the problem statement, entered first and tlie distractor variable 
DIST entered on the fourth step of the regression. The cognitive variables, 
such as memory and order, did not enter as soon or in the same order as 
when students solved problems at a CAI terminal. 

Only the first five variables contributed to an increase in R^ of 
1 percent or more. This is good because it is not practical to take 
into account more than four or five variables when writing word problems, 
even if additional variables were able to account for a larger portion of 
the variance than that indicated thus far. Thus is is important that the 
optimal set of variables be found. The regression equation after the fifth 
step was: 

Zj = -.73 + .02Xq + .19X^g + .22X^3 + .03X^g + .23X23. W 

The raw regression coefficients for the variables X/>-Length, 
Xig-Distractor, Xi8-N0MC2, Xiq-COLC, and Xjo-Division, were .02, J9 , .22, 
.03, and .23 respectively. The percent of the total variance that was 
accounted for by each of the variables in the presence of the other four 
variables, was 21, 11, 32, 1, and 23 for variables Xa, Xw, X^gt X,q, and 
X20, respectively. The percent of increase in variance accounted tor by each 
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step was kSs 3, 26, 1, and. 13 for the variables Xg, Xw, X^g, X^g, and 
X2Q, respectively. The order of entry, by steps, for the variables Xs, 
^16» ^18» ^19» ^20» li '*! 2, 5, and 3, respectively. Comparing 
these results shows now the impartance of a contribution by a variable is 
adjusted in the presence of other variables. These results show why one 
must resist the temptation to use the raw regression coefficients, the 
step at which the variable was entered^ or the increase in variance 
accounted for by each step in determining the importance of the contri- 
bution of each variable in the final equation. 

The final linear regression model described in this section gave a 
surprisingly good account of the difficulty level of a set of verbal 
problems^for fifth-grade students. Five variables were found to account 
for almost 87 percent of the variance in the observed probability correct. 
The variable that accounted for most of the variance was NOMC (32 percent),/ 
the multiplication variable, followed by (JUOT (23 percent), the division 
variable, then LENTH (21 percent), the number of words in the problem 
statement, DISTR (11 percent), the verbal distractor variable, and finally 
COLC (1 percent), the addition-subtraction variable. 

The first follow-up study was intended to replicate the previous 
study using as subjects students in Grades k-S with different achievement 
levels from different schools located in different socioeconomic- level 
communities. It was hoped that any differences among students due to either 
grade level, achievement level, or economic background might b^ evidenced 
by a different order of entry of the variables in the regression analysis 
for each group. The 19 variables and the same set of 30 problems used 
in the follow-up study were the same as those used in the previous study. 

Two of tlie variables which accounted for a large portion of the 
variance in probability correct in the earlier study also accounted for 
a significant amount of the variance In the present study. In addition, 
the number of words In the problem statement, the ability to recall needed 
facts and perform needed conversion of units, and the number of different 
operations were variables that entered as one of the first six steps in a 
stepwise linear regression for the total population. 

The importance of three of the variables, length (p-*.Ol), 
multiplication (p^.Ol) and division (p^.Ol) Is indicated by their level 
of significance in the regression equation for the total group. In terms 
of the results of the follow-up study. It appeared that the variables for 
multiplication, division, length,- recall, conversion, and operations were 
the most Important determinants of word-problem difficulty over all for 
students in Grades when problems were solved using paper and pencil. 

A second follow-up study examined the Influence of the number of words 
in problem statements on error rate. Three forms of the 30-problem set 
used in the other studies were prepared In which the number of words In 
the problem statements were systematically varied, were administered to" 
classes of students in Grades 't-S. 

Three variables which accounted for a large proportion of the variance 
in the first follow-up study also accounted for a significant amount of 
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the variance in the second follow-up study. The three variables were those 
for multiplication, division, and recall. A second set of three variables 
which also entered consistently among the first six in the linear 
regression in the second followup study were those for the number of words 
in the problem statement, conversions, and operations. Of particular 
interest was the variable for the number of words in the problem statement. 
The failure of the length variable to enter the regression consistently 
oyer all forms of the test sets leads one to conclude that it is not 
simply the number of words in the problem statement that influences 
difficulty, but the number of words in. relatlon to other factors. This 
was Indicated in the second follow-up study by the level of significance 
of the length variable on only one form of the test set. Form 2. 

In summary, the six variables listed above, when used in a natural 
statement of a verbal problem statement have been found to account for a 
significant amount of variance in error rate In a series of studies with 
students at different grade levels from different socioeconomic backgrounds. 
The SIX variables used In this study were those for multiplication, 
division, recall, conversions, operations, and the number of words In the 
problem statements. The variables were defined above and were labeled, 
N0MC2, QUOT, Recall, Conversion, Operations, and Length, respectively. 

The purpose of the present study was to determine whether it was 
possible to prepare a set of word problems In arithmetic In terms of the / 
above-mentioned variables and using the regression equation specified 
earlier, for which the predicted level of difficulty was a close 
approximation of the' actual level of difficulty as Indicated by the pro- 
portion of students who were able to solve each exercise. 



Methods 

Subjects 

Three hundred forty students in grades four through nine participated 
in the study; one hundred sixty-one were in elementary school in Pleasant 
Gap, Pennsylvania, and 179 were in junior high school in Bellefonte, 
Pennsylvania. Bel lefonte and Pleasant Gap are small towns in low-middle 
class areas in central Pennsylvania. Two classes each of fourth-, fifth-, 
and sixth-grade students participated In the elementary school and two 
classes each of sevienth-, eighth-, and ninth-grade students participated 
in the junior high school. The. elementary school classes were of average 
ability. One class of seventh grade students was considered a high 
ability group. The other class was an average ability group. All students 
in each class took part in the study. The distribution of students by 
grades and classes is shown in Table 1. 



Insert Table 1 About Here 
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* Table I 

Distribution of Students by Grades 



Class 


Grade 




k 


5 


6 


7 


8 


9 


I 


29 


23 


32 


36 


31 


30 


2 


26 


27 


2f> 


33 


30 


19 


Total 


55 


50 


56 


69 


61 
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Construction of Problem Sets 



Eight sets of arithmetic problem solving exercises, each containing 
twenty word problems were prepared during the month of April, 1972. 
Problem sets l-IV and sets V-VIII were prepared for user in grades four 
through six and seven through nine, respectively. The primary purpose 
for preparing two groups of exercises was to accommodate the higher com- 
putational abilities of students In the upper grades. The basic 
differences in the two groups of exercises were that division exercises 
in sets l-IV had no remainders and the use of fractions was not required; 
whereas, division exercises in sets V-VIM had remainders and computation 
with fractions was required. Problem sets I and IV consisted of problems 
requiring one or two operations for solution. Divisors were all single 
digits with one, two, or three digit quotients. Problem sets II and V 
contained problems requiring one or two operations for solutions. Divisors 
were one or two digits and quotients were one, two, or three digits. 
Problem sets III and VII consisted of problems requiring from one to 
three operations for solution. Divisors were multiples of 10 or single 
digits and quotients were one, two, or three digits. Sets IV and VIII 
contained problems requiring from one to four operations for solution. 
Divisors v/ere one or two digits' and quotients were one, two, or three 
digits. One problem sets I, II, V, and VI problems were randomly 
arranged with respect to operations necessary for solution. On Problem 
sets Illy IV, VII, and VIII, the problems were arranged In order, beginning 
with those which required one operation for solution followed by those 
which required a greater number of operations. The operations required 
for the solution of each exercise In each problem set are given In Table 2. 



Insert Table 2 About Here 



Identical problem sets were used for each of the three elementary 
grades ('*,5, and 6) and for each of the three grades In the junior high 
school (7,8, and 9). Each set of problems^was mimeographed. The first 
page of each test booklet gave d I rections ''to the students for taking the 
problem set and the remaining five pages presented the problems, four per 
page, with work space provided. 

The number of students to whom each problem set was administered is 
shown In Table 3. 



Insert Table 3 About Here 



The sets of problems were administered according to the time schedule 
given In Figure 1 . 



Insert Figure 1 About Here 
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Table 2 

Operations Required In the Exercises 
in Each Set 
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• Table 3 

The Number of Students Taking Each Problem Set 



Grade 

k 
5 
6 

7 
8 

9 



Problem Set 



IV 



52 53 
'»7 ^9 
5** 56 



55 53 
it6 1(9 
'»9 '»6 



V 


VI 


VI 1 


VI 1 1 










65 


6it 


66 


69 


53 


61 


57 


60 


'»3 




50 


• '»9 
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I. Wednesday, May 3, 1972 
Grades A, 5, and 6 

II. Thursday, May 11, 1972 
Grades k,5, and 6 

III. Thursday, May 18, 1972 
Grades A, 5, and 6 

IV. Thursday, May 25, 1972 
• Grades A, 5, and 6 



V. Friday, May 5, 1972 - Grades 7 and 9 
Tuesday, May 9, 1972 - Grade 8 

VI. Tuesday, May 9, 1972 - Grade 7 and 9 
Tuesday, May 23, 1972 - Grade 8. 

VII. Tuesday, May 16, 1972 
Grades 7,8, and 9 

VIII. Tuesday, May 23, 1972 - Grades 7 and 9 
Thursday, June 1, 1972 - Grade 8 



Figure 1. Time Schedule for Administering Problem Sets 
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The reason the eighth grade classes did not always take each problem 
set on the same day as the seventh and ninth grade classes wds that the 
eighth grade classes were on a field trip the day the first problem set, 
Set V, was initially given. 

In the junior high school, the problem sets were administered during 
the regular '♦y-minute mathematics class period. In the elementary school 
the sets were administered at a time that was convenient to the teacher 
and which allowed the students ample time to finish the problems; this 
time was approximately 35^^0 minutes. 

A script was followed by the experimenters while introducing the 
sets of exercises to each class to attempt to standardize the testing 
situation. The text of the script Is included as Appendix A. The 
experimenters answered all questions relevant to procedural difficulties 
and helped to clarify meanings of certain words or phrases bu . gave no 
assistance in solving any of the problems. 

Problem sets I and V are included as Appendix B. 

The eight sets of problems were coded for each of the six variables 
mentioned earlier. A small computer program was written to apply the 
following regression equation to each of the problems in each problem 
set to determine the predicted probability correct for the problem. 

9; = -.96105-0.559X2+0.0227X3+0.218X1^-0.028X5+0. ISSXg+O.ZS'iX^ where 

is the predicted probability correct for the 1 problem In the 
problem set and X2, X^, X/^, X5, Xg, and X7 are the variables RECAL, LENTH 
CONVR, 0PER3, N0MC2, ^nd QUO, respectively. An antllog transform was 
applied to the value of P obtained using the regression equation so 
that the predicted probabi 1 1 ty. correct would be between zero and one. 
The regression equation above was derived from a study by Jerman (1971) 
in which the best over-all fit to the data for Grades k-S was given by 
this equation with the variables listed above. 

Each of the problem sets was corrected by a project staff member. 
Problems which were omitted were not Included In the analysis; i.e., 
the percent correct for a given problem was the percent of those students 
who attempted the problem who got the correct answer. A stepwise linear 
regression computer program, BMD02R, (UCLA), was modified to Include the 
aforementioned log and antllog transforms. 
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Results 

The predicted and observed probability correct for each exercise in 
Problem Sets l-IV pooled over student in Grades are shown In Table h 
along with the chi-square value of each exercise. The predicted probability 
correct was derived from the regression equation which gave the best over- 
Insert Table h about here 

all fit to the data for students in Grades ^-9 In a previous study, as 
mentioned In the previous section. The data from Set I in Table k are shown 
graphically in Figure 2. It is clear from inspection of both Table 3 and 
Figure 2 that the predictions are only fair, at best, in terms of chi-square. 

Insert Figure 2 about here 

The predicted and observed probability correct for each exercise in 
Problem Sets V-Vlll and chi-square is shown In Table 5. The data from Set V 

Insert Table 5 about here 



in Table 5 are shown in Figure 3. It is obvious from inspection of the data 

Insert Figure 3 about here 



presented in Table 5 and Figure 3 that the prediction equation is in need of 
further refinement. The values for R presented In Table 6 for each test at 



Insert Table 6 about here 



each grade level and the total (T) are not low, but neither are they as high 
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as the corresponding values In previous studies. The order of entry of each 
variable is also shown In Table 6. These data do Indicate, however, that the 
model Is still accounting for a significant amount of the observed variance. 

The data from the previous study were reexamined and new regression 
equations for each grade level were computed since the over-all predictions 
were less accurate than hoped. As one might expect, the variables which gave 
the best general account of the observed variance for all grades A-9 In the 
previous study were not necessarily the same set of variables which gave the 
best account of the observed variance at any particular grade leve. Therefore, 
the Problem Sets l-VIII were recoded In terms of the six variables which 
gave the best account of the observed variance is probability correct for 
each grade level In the previous study. The new regression equations were 
used to generate predictions for all four tests administered ^t each grade 
level. These predicted and observed probabilities and chl-square values for 
each exercise on each test, by grade level, are shown In Tables 7-12. It Is 



Insert Tables 7-12 about here 



clear from the magnitude of the chl -square values In each table that the re- 
gression equations gave a generally better predictions In the lower grade levels 
than In the higher grade levels. Figures h, 5, and 6 present graphs of the 
observed and predicted probability correct for each exercise In Set III and 
Set VII for Grades A, 6, and 8 respectively. In this case, the best prediction, 
of the three problem sets graphed appears to be at the sixth-grade level. 
However, It Is nof" yet a satisfactory fit for curriculum development purposes. 

Jnsert Figures 4, 5, 6 about here 
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Two sets of exercises, Problem Sets I and IV, were administered to 
students in Grades A, 5, and 6. The probability correct for each exercise 
on each problem set is shown in Figure 7- Perhaps the most important finding 
'from this set of data is the degree of parallelness of the curves for each 
grade. What seems to be indicated is that the exercises have the same 
relative difficulty for students at each grade level. A similar pattern can 

Insert Figures 7, 8 about here 



be seen in the graphs presented in Figure 8. In Figure 8, the graphs are 
much closer, physically, indicating that the difficulty level for each exercise 
for each of the grades, 7-9, is much the same. Apparently the relative 
difficulty of the exercises was almost exactly the same for^.studehts at the 
several different grade levels. 

The types of operations and the number of steps required to solve each 
exercise was presented In Table 2. As can be seen in Table 2, Problem Sets 
I and V had the same number of each type of exercise. In fact, the problem 
sets were identical In terms of the number of operations required for solution, 
the operation itself, and differed only In vocabulary level and the difficulty 
of the combination used. Problem Sets I and V were the simplest in terms of 
the variables Just mentioned. 

Problem Sets IV and VIM were the most complex in terms of the number 

of steps required for solution, as can be seen In Table 2. The graphs of the 

> 

respective probabilities for each exercise are shown In Figure 9 and 10* 



Insert Figures 9i 10 about here 



Again, there Is a strong indication that the relative difficulty of the 
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exercise remained constant for students at different grade levels. 

The residuals tend to give a somewhat clearer picture of the degree of 
accuracy of the prediction. The residuals are presented In Tables 13-18. 
The mean of the residuals is shown at the bottom of eadi cdlumn. As can be 



Insert Tables I3-I8 about here 



seen, the mean residual per cent for each problem set Is fairly small. The 
range Is from h per cent, Table 15, to 15 per cent, Table 18. The finding 
that the mean residuals are as low, overall II per cent, Is quite encouraging 
for a first effort. What one world like to be able to do Is to reduce the 
residual to less than 5 per cent. Perhaps, In future efforts, this will be 
possible. 
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Table 13 

Residual Per Cent Correct For Each 
Problem Set> Grade k 



Problem 


Set 1 


Set II 


Set III 


Set iV 


1 


.002* 


.270 


.268 


.285 


2 


.061 


.100 


.026 


-.035 


3 


-.126 


.137 


-.03k 


.Oi»9 


i» 


.032 


.]2k 


-.00i» 


.03'* 


5 


.168 


.053 


.2i»9 


.3'»3 


6 


.0k2 


.063 


-.058 


.0i»2 


7 


-.190 


.112 


.101 


-.161* 


8 


.018 


.05'* 


-.i»22 


-.195 


9 


-.09'* 


.001 


-.010 


-.190 


10 


-.000 


.167 


.032 


.009 


11 


JOk 


.261 


.1i(8 


-.0'»5 


12 


.082 


.03'* 


-.085 


-.000 


13 


-.000 


.0'»2 


-.027 


-.07'* 


1i» 


-.I8i» 


.002 


-.01^ 


-.119 


15 


.283 


.065 


-.015 


-.002 


16 


.028 


.116 


.007 


.056 


17 




.021 


.103 


-.021 


18 


.023 


.oko 


-.050 


.131 


19 


.162 




-.053 


-.019 


20 


.052 


.078 


.008 


- .050 


T 


.09** 


.09 


.08 


.09 



• 2 per cent 
9 per cent 
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Table ]k 



Residual Per Cent Correct For Each 
Problem Set, Grade 5 



r rob 1 em 


Set 1 


Set 1 1 


Set III' 


Set IV 


1 


• 100* 


.110 


-.030 


.080 


Z 


1 O f I 

. 1 2h 


.08] 


-.019 


.091 


3 


-.239 




.103 


.01 1 




-.0Z3 


.OoH 


All 

-.01 1 


. 119 


5 


.0/2 


.oks 


-.00^ 


1 t 1 
. iHl 


6 


-.122 


-.296 


-.099 


.002 


7 


- . 1 29 


- 007 


167 




8 


.012 


-.010 


-.370 


-.130 


9 


-.280 


.U1 


.061* 


-.083 


10 


-.000 


-.126 


.Oh3 


.121 


11 


.030 


-.116 


.160 


-.085 


12 


-.029 


.101 


-.131 


.000 


13 


.027 


-.186 


-.1'»9 


-.0'»6 


]h 


-.184 


-.068 


-.117 


-.136 


15 


.091 


-.007 


.227 


.163 


16 


.100 


.25'* 


.061 


-.027 


17 


.075 


.07'» 


.1'>6 


-.156 


18 


.226 


-.132 


.067 


.271 


19 


. .166 


-.239 


-.135 


.173 


20 


.003 


.286 


-.0^3 


-.222 


T 


.10 . 


.12 


.10 


.12 



*. 10 per cent 
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Table 15 



Residual Per Cent Correct For Each 
Problem Set> Grade 6 



Problem 


Set 1 


Set II 


Set III 


Set IV 


1 


.109* 


.104 


.049 


.059 


2 


.099 


.076 


.033 


.08^ 


3 


-.081 


.048 


.028 


.03A 


k 


-.020 


.023 


.058 




5 


.098 


.023 


.051 


.054 


6 


-.I8i» 


-.163 


-.051 


.021 


7 


-.191 


- 9nA 


. u 1 0 


9 1 Q 


8 


.063 


.056 


-.220 


-.091 


9 


-.332 


.083 


. -.005 


-.054 


10 


-.000 


-.050 


-.037 


.061 


11 


.062 


-.208 


.084 


-.186 


12 


.002 


.171 


.014 


-.009 


13 


.032 


-.008 


-.003 


.036 


14 


-.12i» 


.010 


-.084 


-.320 


15 


.078 


-.256 


-.013 


.120 


16 


.062 


.207 


.040 


.114 


17 


-.06i» 


.052 


-.010 


. -.046 


18 


.138 


-.121 


.010 


.295 


19 


.065 


-.299 


-.129 


.100 


20 


.070 


.129 


.002 


-.236 


T 


.09 


.11 


.04 


.11 



* 10.9 per cent 
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Table 16 

f Residual Per Cent Correct For Each 

h Problem Set, Grade 7 



Prob 1 em 


Set V 


Set VI 


^0 1* MX \ 
u VII 


oet V 1 1 1 


1 


-.006* 


- 272 






2 


-.058 




inc 
• 1 


_ 1 1 n 
-.110 


3 


-.306 


.022 


1 (;i 




it 


114 




1 fio 

• 1 0^ 


.010 


5 


.141 


.276 


• i 1 D 


.U7U 


6 


.063 


- "^28 

• ^£m\J 


• yj j\j 




/ 




.037 


.048 


-.136 


8 


.136 


.000 


-.111 


.015 


9 


-.095 


.120 


.069 


-.182 


10 


.000 


-.062 


-.335 


-.444 


11 


.019 


.296 


-.043 


-.300 


12 


-.057 


.039 


.085 


.036 


13 


-.143 


.031 


-.141 


-.170 


14 


.071 


-.133 


.136 


-.115 


15 


-.058 


.084 


-.055. 


.050 


16 


.074 


-.318 


-.321 


.200 


17 


.091 


.091 


-.020 


.204 


18 


-.343 


-.026 


.013 


.073 


19 


.004 


MS 


-.204 


-.037 


20 


.092 


-.218 


.027 


.039 


T 


.10 


.12 


.11 


.14 



* -.6 per cent 
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Table 17 



Residual Per Cent Correct For Each 
Problem Set» Grade 8 



Problem 


Set V 


Set VI 


$&t VII 


Set VIII 


1 


.099* 


"-.16'» 


.057 


.119 


2 


.015 


-.081 


.117 


-.022 


3 


-.37'» 


-.058 


.197 


.m 


k 


.013 


.Ol** 


.32'* 


-.0'»6 


5 


.108 


.337 


.060 


.0'»7 


6 


.067 


-.297 


-.012 


.<«05 


7 


-.281 


.063 


.088 


- .1 10 


8 


.133 


.000 


-.186 


-.036 


9 


-.036 


.102 


.153 


-.2'»5 


10 


.020 


-.171 




-.'♦02 


11 


.0<«6 


.32f 


.021 


-.195 


12 


-.l'»9 


.060 


.163 


.082 


13 


-.198 


.069 


-.005 


-.109 


1<« 


.038 


-.230 


.06<« 


-.080 


15 


-.089 


-.06'* 


-.116 


.0'»9 


16 


.173 


'MS 


-.317 


.2'»7 


17 


.053 


-.036 


-.030 




18 


-.2'»9 


-.090 


.118 


.050 


19 


.09'* 


.056 


-.228 




20 


.011 


-.018 


-.085 


.0<(6 


X 


.11 


.13 




.13 



* 9.9 per cent 
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Table 18 



Residual Ber Cent Correct For Each 
Problem Set^ Grade 9 



Problem 


Set V 


Set VI 


Set VI 1 


Set VI 1 1 


1 


.100* 


-.3'»6 


.157 


.llt8 


2 


M 


.037 


.151 


.030 


3 


-.268 


-.132 


.130 


.238 


li 


.13'* 


-.006 


.157 


-.116 


5. 


.167 


.371 


.200 


.135 


6 


.036 


-.278 


-.13'» 


.276 


7 


-.'♦19 


.058 


.112 


- 1 


8 


.19'* 


.000 


-.170 


-.030 


9 


-.127 


.032 


-.0'»5 


-.113 


10 


-.096 


-.006 


-.371 


-.'♦22 


11 


.0'»9 


.286 


-.066 


-.221 


12 


-.098 


.06'« 


.I'll 


-.126 


13 


-.185 


.087 


-.036 


-.110 


]k 


.107 




.071 


-.156 


15 


-.058 


.091 


-.02l» 


.17'» 


16 


.158 


-.283 


-.201 


.261 


17 


.0'«6 


AkO 


-.010 


.IS'* 


18 




.022 


-.039 


-.010 


19 


.071 


.002 


-.217 


-.08'* 


20 


-.037 


-.07? 


.OOif 


.08*^ 


T 




.13 


.12 


.15 


* 10.0 


per cent 
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Discussion - Conclusions 

Although several studies have attempted to define structural variables 
that are capable of accounting for a significant amount of the observed variance 
In the probability correct of word problem exercises In arl thmetic, to the 
writer's Icnowledge this study has been the first to attempt to predict 
difficulty probability correct, in advance. ' 

Eight sets of word problem exercises were prepared in which such things 
as the number of words used, the number of steps required for solution, the 
operations required, and the difficulty of the computations Involved were 
carefully controlled. The problem exercises were coded in terms of the 
variables found to account for the most variance in observed probability 
correct in previous a study. The regression analysis did verify that the 
six variables which accounted for the most variance In the pooled data for 
Grades ^1-9 in the previous study a 1 so accounted for a significant amount of 
the variance in the present study. The predictions, however, were not as 
accurate as It was anticipated they would be in terms of a chl-square test. 
Even when separate regression equations were used to predict the probability 
correct for each individual grade level, the predictions were not as accurate 
as anticipated although they were somewhat better than for the general case. 
The actual magnitude of the residuals does not present such a dark picture. 
In fact. It was encouraging to observe that the over all mean residual was 
11 per cent, range 4-15 per cent. 

It was also encouraging to find that the relative difficulty of the 
word problem exercises was constant over grade levels. This lends support 
to the assumption that there must be a set of variables which can be fourtd 
to account for the observed consistent levels of difficulty over grades. 
That Is, because those graphs are as they are, lending support to earlier 
findings, we believe we are correct In following the approach In this study. 

« " 50 



What remains to be done is to Improve the set of variables and thereby 
improve the predictive power of the regression equations. A small set of 
Independent structural variables will be defined. The scope will be broadened 
to Include syntactic variables which should account for some of the complexity 
In the language of the word problem itself. Together, the computational and 
syntactic structural variables may provide. the key to unlocking a complex 
situation. 

It is recognized that the set of variable used In this study Is In need 
of further refinement. However, for the purposes of this project, the objec- 
tives have been met. That Is, It does seem possible to use structural vari- 
ables to predict the relative difficulty of word problems in arl thmetic with 
a reasonable degree of accuracy. We are encouraged, but much work remains 
to be done . 
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Appendix A 
Administering The Problem Sets 

Be pleasant - smi le 

Introduce yourself in something like the following manner: 

Good morning, I am from Penn State University at 

Penn State, One of the things we are trying to do is to find out 
what makes word problems so difficult for students to learp to solve. 
We have already found several clues and hope to find more, with your 
help. 

I am going to give each of you a set of word problems to solve. 
We think we know which ones are easy and which are difficult because of 
our research. Now I want you to work these for us so that we wi 1 1 
know whether or not we have found the right clues. 

All you need is your pencil. It is important that you try every 
problem. Do your best on each problem. If a problem Is too difficult, 
go on to another and come back to it later if there Is enough time. 

We want you to put your name on the first page so that if you 
are able to solve a problem no one else can solve, we would like to 
know how you did it. Also, if you are not able to do a problem every- 
one else can do, we may want to get you to tell us what you thought was 
difficult about the problem. Do all your work on the paper I give you. 
If you finish ahead of time, turn your paper over. I wl H collect 
them when everyone is finished. 

Any questions? 

Alright, as soon as I hand out the papers you may begin. 

'*7 
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jup.i^ti p?r s:onTl Thft car wan a n^^^ n.u i."<* wUh a v?iv/l top.. 



t 

I. / 



{ic/./ •T«;>r;v -ilv; I'.u catch ihM :. 

f5»;:; in the r4>rTfhin and r. 



<^r> i:t^f>r' l^-or;;^^ how many bo.^ri ^ i -^?;:: ii i;': ..-JJ : 



af^f an;! c.-'h .-..v? R.JSes durSnfi t'i>-;-. i.';<. :r .v.; • .jt^ j^^-'mip Mi't: uffo ' v- 
ii».-«.'iy K>ri!';: ^i;-' U travel Sn shn f)f;:l. ::>^.,;;f.,'; :; ? 



?9o 

for i> >.'.iC;;:lMi> wJnJo^/,, What r-Afi • • ^;f«f7 
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